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(57) Abstract 

A microscope (12) is used to optically section an object at a plurality of focal depths over a plurality of time periods The optical 
sections are digitized (18), and a tag allows identification of at least the time and the focal depth of each digitized optical section Image 
processing creates an outline of the periphery of the object through application of a complexity threshold algorithm A plurality of 
parameters representing the motility and morphology of the object are calculated. A three dimensional graphical representation of the object 
is reconstructed from the plurality of digitized optical sections for computerized viewing (26). 



FOR THE PURPOSES OF INFORMATION ONLY 



Codes used to identify States party to the PCT on the front pages of pamphlets publishing international applications under the PCT. 



AL 


Albania 


ES 


Spain 


LS 


Lesotho 


SI 


Slovenia 


AM 


Anncnia 


FI 


Finland 


LT 


Lithuania 


SK 


Slovakia 


AT 


Austria 


FR 


France 


LU 


Luxembourg 


SN 


Senegal 


AU 


Australia 


GA 


Gabon 


LV 


Latvia 


sz 


Swaziland 


AZ 


Azerbaijan 


GB 


United Kingdom 


MC 


Monaco 


TD 


Chad 


RA 


Bosnia and Herzegovina 


GE 


Georgia 


MD 


Republic of Moldova 


TG 


Togo 


BB 


Barbados 


GH 


Ghana 


MG 


Madagascar 


TJ 


Tajikistan 


BE 


Belgium 


GN 


Guinea 


MK 


Tlte former Yugoslav 


TM 


Turkmenistan 


BF 


Burkina Faso 


CR 


Greece 




Republic of Macedonia 


TR 


Turkey 


BG 


Bulgaria 


HU 


Hungary 


ML 


Mali 


TT 


Trinidad and Tobago 


BJ 


Benin 


IE 


Ireland 


MN 


Mongolia 


UA 


Ukraine 


BR 


Brazil 


IL 


Israel 


MR 


Mauritania 


UG 


Uganda 


BY 


Belarus 


IS 


Iceland 


MW 


Malawi 


US 


United States of America 


CA 


Canada 


IT 


Italy 


MX 


Mexico 


uz 


Uzbekistan 


CF 


Central African Republic 


JP 


Japan 


NE 


Niger 


VN 


Viet Nam 


CG 


Congo 


KE 


Kenya 


NL 


Netherlands 


YU 


Yugoslavia 


CH 


Switzerland 


KG 


Kyrgyzstan 


NO 


Norway 


zw 


. Zimbabwe 


CI 


Cfltc d'lvoirc 


KP 


Democratic People's 


NZ 


New Zealand 






CM 


Cameroon 




Republic of Korea 


PL 


Poland 






CN 


China 


KR 


Republic of Korea 


PT 


Portugal 






CU 


Cuba 


KZ 


Kazakstan 


RO 


Romania 






CZ 


Czech Republic 


LC 


Saint Lucia 


RU 


Russian Federation 






DE 


Germany 


LI 


Liechtenstein 


SD 


Sudan 






DK 


Denmark 


LK 


Sri Lanka 


SE 


Sweden 






EE 


Estonia 


LR 


Liberia 


SG 


Singapore 







WO 99/67739 PCT/US99/13193 

THREE DIMENSIONAL DYNAMIC 
IMAGE ANALYSIS SYSTEM 

Related Applications 

5 The present application claims priority based upon Provisional Application 

60/051,095 field June 27, 1997. 

Background of the Invention 

10 The invention relates generally to motion analysis, and more specifically, to a three 

dimensional elapsed time system for the analysis of the motility and morphology of a moving 
object. 

The analysis of the behavior of motile, living cells using computer assisted systems 
comprises a crucial tool in understanding, for example, why cancer cells become metastic, 

15 why HIV infected cells do not perform their normal functions, and the roles of specific 

cytoskeletal and signaling molecules in cellular locomotion during embryonic development 
and during cellular responses in the immune system. Further, motion analysis systems have 
been used to analyze the parameters of shape and motion of objects in a variety of diverse 
fields. For example, such systems have been used for analysis of such diverse dynamic 

20 phenomena as the explosion of the space shuttle Challenger, car distortion, echocardiography, 
human kinesiology, insect larvae crawling, sperm motility and bacterial swimming, to 
analyze cell movement and moipho logical change, to quantize shape changes of the 
embryonic heart, to quantize breast movement for reconstructive surgery, and to analyze 
human form and movement. Often times the information required to analyze such systems 

25 required manual gathering. For example, in analyzing embryonic heart action, a researcher 
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would display an echocardiograph of a heart on a monitor and make measurements of the 
monitor using a scale, or the like, held up to the screen. The tedious and time consuming 
nature of these types of manual measurements severely limit the practicality of. such an 
approach. 

5 A motion analysis system for the biological study of cell motility and morphology 

requires the use of state of the art videomicroscopy, image processing, and motion analysis 
systems. United States Patent No. 5,655,028 describes a computer assisted two dimensional 
system for analyzing the dynamic morphology of moving objects, and particularly of 
organisms on the cellular level. An electronic signal corresponding to the images, for 

10 example, from a video camera, is input into a digitizer which identifies the coordinates of the 
periphery of the mobile object in each of the images. A digital processor processes the 
contour information, and a computer controlled by a software program having image 
'■■ processing and graphics capabilities calculates a plurality of parameters representative of the 
shape and motion of the object. The output from the computer may be displayed in a 

15 graphical representation, tabular form, in the formation of animations on a monitor, in a hard 
copy print out of tables, or animations and other graphical representations in two dimensions. 
Such a system, however, lacks the ability to fully capture every aspect of the dynamic 
morphology of a moving object. For example, the analysis of the motion of certain types of 
amoebae and human leukocytes translocated across a substratum revealed that these objects 

20 changed area in a cyclical fashion when viewed in a two dimensional focal plane. In other 
words, portions of the cells go in and out of focus over time. Additionally, in some cases 
pseudopods can form off of the substratum. These observations indicate that only a three 
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dimensional motion analysis system can fully capture the dynamic morphology of these types 
of moving objects. 

Consequently, a need exists to develop a 3-D motion analysis system for capturing the 
motility and morphology of moving objects, particularly on the cellular level, that obtains 
5 optical sections of a moving object within a time interval short enough so that the motility of 
the object does not significantly affect each static 3-D reconstruction, to reconstruct a 3-D 
mathematical model of the image representative of the object within short time intervals, and 
to repeat the reconstruction process at short time intervals in order to achieve three 
dimensional elapsed time analysis of the motility and morphology of the moving object. 
10 The specification of the invention includes a microfiche appendix submitted according 

to 37 C.F.R. §1.96 of twenty-two (22) microfiche, comprising a total of 2139 frames. The 
appendix is a printout of the source code of the computer program which controls operation 
of the present invention. - 

Summary of the Invention 
An object of the present invention comprises providing a method for the three 
dimensional elapsed time analysis of the motility and morphology of a moving object. 

A further object of the present invention comprises providing a system for the three 
dimensional elapsed time analysis of the motility and morphology of a moving object. 
These and other objects of the present invention will become apparent to those skilled in the 
art upon reference to the following specification, drawings, and claims. 

The present invention intends to overcome the difficulties encountered heretofore. To 
that end, a microscope is used to optically section an object at a plurality of focal depths over 

3 
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a plurality of time periods. The optical sections are digitized, and a tag allows identification 
of at least the time and the focal depth of each digitized optical section. Image processing 
creates an outline of the periphery of the object through application of a complexity threshold 
algorithm. A plurality of parameters representing the motility and morphology of the object 
are calculated. A three dimensional graphical representation of the object is reconstructed 
form the plurality of digitized optical sections for computerized viewing. 

Brief Description of the Drawings 
Fig. I is a component diagram of a 3-D digital image analysis system. 
Fig. 2 is a top plan view of the optical sectioning and outlining of a motile object at a 
plurality of focal depths. 

Fig. 3 is a block diagram of the outlining process. . 

Fig. 4 is a top plan view of an optical -section of an object and the digitized outline of 
the object. 

Fig. 5 is a dilation of the digitized outline of Fig. 4. 
Fig. 6 is an erosion of the digitized outline of Fig. 5. 
Fig. 7 is a top plan view of a digitized optical section. 

Fig. 8 is a display from a graphical user interface depicting a plurality of parameter 
representing the motility and morphology of an object. 

Fig. 9 is a top plan view of a plurality of optical sections and their corresponding 
outlines at a plurality of focal depths. 

Fig. 10a is a top plan view of a plurality of digitized optical sections at a plurality of 
focal depths, with the out of focus backgrounds subtracted out. 
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Fig. I Ob is a stacked image reconstruction of the plurality of optical sections shown in 
Fig. 10a viewed from a plurality of attitudes. 

Fig. 10c is a faceted image reconstruction of the optical sections of Fig. 10a viewed at 
a plurality of attitudes. 

Fig. 1 1 is a top plan view of outlines of a plurality of digitized optical sections stacked 
with contour levels corresponding to focal depths. 

Fig. 12 is a stacked image reconstruction with a slotted interior viewed over several 
periods of time, with a faceted image reconstruction of the portion of the object 
corresponding to the slot. 

Fig. 13 is a top plan view of an outline of a digitized optical section and a slot. 

Fig. 14 is a slotted faceted image reconstruction of an object over several periods of 
time. Fig. 15 is an elevation view of the outline of a plurality of digitized optical sections 
with a lateral indentation. . 

Fig. 16 is a component diagram of an alternative 3-D image analysis system. 

Fig. 17 is an illustration of turning angles used to compute convexity and concavity. 

Fig. 18 is a graph of speed versus time. 

Fig. 19 is a top plan view of a digitized optical section and outline. 
Fig. 20 is a top plan view of a digitized optical section and outline. 
Fig. 21 is a top plan view of a digitized optical section and outline. 
Fig. 22 is a top plan view of a digitized optical section and outline. 
Fig. 23 is a top plan view of a digitized optical section and outline. 
Fig. 24 is a top plan view of a digitized optical section and outline. 
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Detailed Description of the Invention 
. In the drawings, Fig. 1 shows a 3-D digital image analysis system (DIAS) 10. The 3- 
D DIAS System 10 comprises an inverted compound microscope 12 fitted with differential 
5 interference contrast (DIC) optics, a camera 14 ; a VCR 16, a frame grabber 18. a character 
generator 20, a computer 22 having a serial port 24, a computer display terminal 26, and a 
key board 28. Additionally, a stepper motor 13 attaches to a focus knob 1 1 of the DIC 
microscope 12. In the preferred embodiment of the invention the stepper motor 13 comprises 
a computer programmed MicroStepZ3D stepping motor. The camera 14 configures for 

10 NTSC video, and in the preferred embodiment of the invention comprises a cooled CCD 
camera which can handle 30 frames per second without motion blurring. The VCR 16 
comprises a conventional high quality tape recorder or video disk system, equipped with a 
frame grabber 1 8. In the preferred embodiment of the invention the frame grabber 1 8 
configures for use with a Macintosh operating system based computer capable of grabbing 30 

15 frames per second of at least a 3/4 size image and storing the results as a QuickTime movie. 
The computer 22 comprises a Macintosh computer, in particular a power PC based computer 
with a core processor speed of at least 225 megahertz, a two gigabyte hard drive, and forty- 
eight megabytes of RAM. The computer display terminal 26 is capable of pseudo three 
dimensional viewing through stereo graphics "crystal eyes" 3-D display screen with special 

20 glasses 29, or at least a fifty percent reduction in resolution and a standard color display with 
inexpensive red blue stereo glasses. 

Those skilled in the art will appreciate the possibility of various changes and 
substitutions to the components of the 3-D DIAS System 10 without departing from the scope 

6 
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of the present invention. For example, the computer 22 can comprise any number of types 
and varieties of general purpose computers, or a digital camera with a direct link to the 
computer 22 could replace the camera 14 and VCR 16. Additionally, the preferred 
embodiment of the present invention utilizes differential interference contract microscopy. 
5 DIC optics has the advantage of high resolution microscopy, without the use of dyes or lasers, 
which may lead to the premature death of the organisms due to increases in heat and the 
effects of phototoxicity. Premature death leads to shortened periods of motility and dynamic 
morphology for analysis. By contrast, confocal optical systems, that use lasers, typically 
require application of stains or dyes to the motile objects. This will kill a living object which 

10 eliminates the possibility of analyzing the objects motility and morphology. Deconvolution 
methods involve phase or standard light microscope images, and presently do not exhibit 
sufficient optical quality to practice the present invention. In other words, while DIC 
microscopy comprise the preferred. method of practice of the present invention, the possibility 
exists to use other microscopy techniques despite their drawbacks. 

1 5 The computer 22 performs the methods of the present invention under computer 

control through the use of programming means in the form of a 3-D DIAS software package 
(see microfiche appendix). The method begins by placing the sample object on the DIC 
microscope 12. Since typically the object comprises a living cell, the object is contained in a 
fluid filled viewing chamber (not shown). Accordingly, the supporting materials must be the 

20 correct width and chemical nature (glass vs. plastic vs. quartz) to be compatible with the focal 
depth and the light transmission for the particular objects used. Magnification must be 
selected which is compatible with the speed of cellular translocation, over a period of 
recording, and most importantly compatible with the size of the cell. The stepper motor 13 
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must be programmed so that one cycle spans the desired Z-axis focal depth. For example, for 
amoebae cells like Dictyostelium amoebae or polymorphonuclear leukocytes, which average 
• fifteen microns in length and usually no more than ten microns in height, a Z-axis distance of 
ten to twenty microns is sufficient but the exact Z-axis distance must be empirically defined. 
5 Next, the method comprises optically sectioning an object at a plurality of focal 

depths over a first period of time. In order to accomplish this optical sectioning a scan rate 
must be chosen. A two second scan in either direction up or down including 30 optical 
sections is more than sufficient for the analysis of cells moving at velocities of seven to 
twenty microns per minute. This rate results in relatively small errors due to cell movement 

10 during the time of sectioning. Since reconstructions can be made from a scan up as well as 
down the Z-axis, a fast rate and a fast frequency of scanning would include sequential up and 
down scans each including 30 frames over one second through ten microns. Although the 
. optical sections can be read.dircctly into the frame grabber 18, it is more effective initially to 
make a video recording or tape for several reasons. First, image acquisition on video tape is 

15 relatively limitless and inexpensive and, therefore, will accommodate extended recording 

periods. Real time frame grabbing will have storage limits. Second, the image acquisition on 
tape allows the character generator 20 and the stepper motor 1 3 to notate each video frame for 
time, height, and direction of scan. As each of the plurality of optical sections are read the 
image from the camera 14 transfers to the VCR 16 then to frame grabber 18, and into the 

20 computer 22 via serial port 24. This process repeats for a plurality of focal depths over a first 
period of time. The focal depth varies through movement of the step remoter 13 fixed to the 
focus knob 1 1 of the DIC microscope 12. The frame grabber 18 digitizes each of the plurality 
of optical sections and then transfers the data to the computer 22. Simultaneously the stepper 

8 
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motor 13 and the character generator 20 transfer information to the computer 22, that 
associates a tag with each of the plurality of digitized optical sections. The tag allows 
identification of at least the time and the focal depth corresponding to each of the plurality of 
digitized optical sections. 

.5 The data transfers into the computer 22, preferably a Macintosh computer, and results 

in the creation of a QuickTime movie. The present invention also works with PICT stacks in 
addition to QuickTime movies. Digitized optical sections can be read into the computer 22 at 
a maximum rate of thirty frames per second or if desired a lower rate such as ten or twenty 
frames per second. Those of ordinary skill in the art will appreciate the applicability of the 

10 present invention to even higher rates of capture, as the technology develops. A twenty 
minute segment read in at thirty frames per second will take more than five hundred 
megabytes of storage on a hard disk. The QuickTime movie is synchronized to the automatic 
up and down scans and the time of the scans are recorded in a synchronization file in the 
computer 22. The frames of the QuickTime movie are then extracted into a 3-D DI AS movie 

15 format, from which the user can select a number of digitized optical sections to be used in 
reconstructions, the interval between reconstructions, and image averaging. For instance, the 
user may only need every other section for reconstructions. The desired digitized optical 
sections are then stored in a compact 3-D DIAS movie file. All subsequent 3-D DIAS 
-procedures access this movie file format. The QuickTime movie format is designed for 

20 smooth viewing in real time and provides a very slow direct access time of two seconds per 
frame. The 3-D DIAS movie currently provides direct frame access at a rate of five frames 
per second, which is presently ten times faster than the QuickTime movies. In addition, if the 
area of the object in a frame takes up a minority of pixels the 3-D DIAS software performs a 



SUBSTITUTE SHEET (RULE 26) 



WO 99/67739 PCT/US99/13193 
compression to reduce memory. Alternatively during the optical sectioning a user can reduce 
the size of the optical section to a specific window which contains only a portion of interest, 
thereby reducing the amount of digitized information. The 3-D DIAS movie allows for frame 
averaging to reduce background noise and accentuate the periphery of the object. For 
5 instance, at a rate of thirty frames per second, every three frames can be averaged in an 

overlapping fashion, resulting in the second to twenty-ninth optical section averaged with the 
two neighboring sections, and the two end sections (one and thirty) average with only one 
neighboring section. 

Fig. 10a shows a portion of a set of twelve digitized optical sections 32 of a 
1 0 Dictyostelium amoebae at one micron increments taken in a two second period and averaged 
over three frames, providing in focus perimeters amenable to subsequent automatic outlining 
(see also Fig, 2). 

. After optically/sectioning the object at a plurality of focal depths over several periods 
of time, and digitizing the plurality of optical sections 32, the next step comprises outlining 
1 5 the periphery of the objects for each of the plurality of digitized optical sections 32. 

Since a typical twenty minute recording of a translocating cell, for example, in which 
thirty optical sections are performed in two seconds and repeated every five seconds, would 
include seven thousand two hundred optical sections, the automatic outlining program is 
clearly essential. For very high resolution reproductions, manual outlining is sometimes 
20 required, but the automatic feature of the present invention is highly efficient and can be 
rapidly edited if necessary, by the user. 

Fig. 2 shows the before and after effect of outlining an object at a plurality of focal 
depths. Fig. 2a shows the original digitized optical sections 32 of an object at twelve 

10 
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different focal depths, and Fig. 2b shows the same digitized optical sections 32 with the 
corresponding outlines 38 included. The outline 38 attempts to trace the circumference of the 
in focus portion of the object. Fig. 2 shows that not only the size of the. in focus portion of 
the object varies at different focal depths, but the surrounding background also varies. This 
5 comprises the significant challenge to the outlining process. In some portions of the digitized 
optical section 32 the boundary between the in focus portion and the out of focus portion 
represents a bright area, in other parts of the digitized optical sections 32 the boundary 
between the in focus and out of focus area represents a dark area. This means that a simple 
gray level thresholding technique, which selects or deselects pixels based solely on their 
10 grayscale value, cannot successfully perform the task of outlining the digitized optical 
sections 32. The present invention uses a combination of a variety of image processing 
techniques to accomplish the task of outlining the periphery of the digitized optical sections 
32. . . ... :.. ...... .' . . . 

Fig. 3 shows in block diagram form the theoretical steps of the outlining process. 
15 Those of ordinary skill in the art will appreciate the fact that the order of the steps depicted in 
Fig. 3 can vary without departing from the intended scope of the present invention, and in 
some cases the computer 22 can perform the steps simultaneously. 

Fig. 3 shows a smooth image step 102. which normally occurs at the beginning of 
image processing, to prepare the digitized optical section 32 for the actual outlining. 
20 Smoothing tends to remove the jagged and rough edges, and reduces the overall contrast. 
The smooth image step 102 involves standard smoothing techniques 

The next step comprises the complexity threshold step 104. Complexity, in this case, 
is defined as the standard deviation from a mean pixel grayscale value within a 3x3 or 5x5 

11 
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pixel neighborhood surrounding the pixel under analysis. The neighborhood is referred to as 
a kernel. Since the perimeter of a cell represents a boundary of high contrast, the standard 
deviation of the grayscale of a pixel at an edge, and the pixels on either side (inside and 
outside of the cell) will be high. Therefore, the complexity will also be high. In other words, 
5 for each of the digitized optical sections 32 the transition between the in focus region and the 
out of focus region is defined by an area of high grayscale contrast. In this manner, 
examining a 3x3 or 5x5 kernel and calculating the standard deviation of the grayscales of the 
kernel allows for identifying the boundaries of the cell periphery for a particular digitized 
optical section 32 at a particular focal depth. For each pixel, based on the pixel's 

1 0 corresponding kernel, a standard deviation representing the amount of grayscale variation 
within the kernel is calculated. A threshold value allows selecting only those pixels with a 
complexity value above the threshold. Thus, kernels with a high standard deviation represent 
areas of high, complexity based on.a large amount of contrast.in that pixel neighborhood. 
Conversely, kernels of low standard deviation represent areas of low complexity due to the 

15 minimal amount of grayscale contrast. This process effectively deselects the background of 
the image and also the interior of the object, since these regions of the digitized optical 
sections 32 tend to exhibit low contrast. The actual threshold value can correlate to a 
grayscale level between 0 and 255, or a percentage between 0 and 100, with the low value 
representing regions of low complexity and the high value representing regions of high 

20 complexity, or any other similar designation. Regardless of the specific designation, the 
threshold represents a cut off level all of the pixels whose kernels yield complexity levels 
below the threshold and these pixels receive a grayscale value of 255 (white). All of the 
pixels with complexity values above the threshold receive a grayscale value of 0 (black). For 

12 
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analysis purposes, therefore, the particular digitized optical section 32 converts to an image 

where the background and the cell interior appears white and only the periphery of the object 

appears black. The black areas then form the outline 38. Typically, increasing the 

complexity threshold value will shrink or remove the outline 38, while lowering the 

5 complexity threshold value will increase the area of the outline 38. 

In most situations further image processing will prove necessary to complete the 

outlining process. In some instances high complexity regions may exist in the background 

areas far outside of the periphery of the object. In these circumstances, simply applying a 

complexity thresholding technique will not remove these regions. Additionally, another 

10 problem that can occur involves the fact thai some regions of the periphery of the digitized 
optical sections 32 do not comprise areas of high complexity. For example, Fig. 4a-b show 
two digitized optical sections 117, 1 18 in which application of the complexity threshold did 
not. form complete outlines 120, 122. In Fig. 4a, the digitized optical section 1 17 appears in 
two sections with a fuzzy low contrast transition between the two. Therefore, application of a 

15 complexity threshold did not properly outline the transition area (see outline 1 17 of Fig. 4b). 
Similarly, the digitized optical section 1 18 shows that a portion of the periphery comprises a 
fuzzy low contrast region, which an application of the complexity threshold technique failed 
to fully outline (see outline 1 18 of Fig. 4a). Accordingly, the outlines 120, 122 in Fig. 4b 
require further image processing. 

20 To deal with the situation of incomplete and partial outlines the 3D-DIAS System 10 

provides the ability to dilate, erode, and smooth the digitized optical sections 32. Referring 
again to Fig. 4, applying the complexity threshold step 104 to digitized optical section 117 
produces outline 120. Similarly, applying the complexity threshold step 104 to digitized 
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optical section 118 produces outline 122. Fig. 4 shows that both the outlines 120, 122 do not 

completely enclose their respective objects 116, 118. The first step in completing the outlines 

120, 122 comprises the dilate step 106 (Fig. 3). Dilation involves selecting every pixel that 

surrounds a black pixel and converting that pixel to a grayscale of 0 (black). Fig. 5a shows 

5 the dilation process applied to the outlines 120, 122. This produces dilations 124, 126, or a 

broader outline that fills in the gaps in the original outlines 120, 122 of Fig. 4b. In particular, 

dilation involves adding the four horizontal and vertical neighboring pixels for each pixel of 

the digitized outlines 120, 122 appearing in Fig. 4b. The dilation process fattens the object by 

the amount of dilation. In this manner, the gaps that appeared in the original outlines 120, 

10 122 fill in. Next, the outer perimeter of dilation 124 and dilation 126 are outlined creating a 
dilated outline 128 and a dilated outline 130 shown in Fig 5b. 

The 3D-DIAS System 10 utilizes additional image processing to smooth the black 
pixels remaining after the dilate step 106. Fig. 3 shows the smooth outline step at 108. Again,- 
the smooth outline step 108 utilizes standard smoothing techniques. For example, one 

15 smoothing technique involves converting the locations of all non-white pixels to a floating 
point number, and then averaging the pixel locations for a neighborhood. Then, a pixel is 
added at the a location as close as possible to the average location. This reduces the 
roughness of the outline 38. 

Application of a grayscale threshold step 1 10 can further enhance the image 

20 processing. The grayscale threshold step 1 10 merely removes pixels with grayscale values 
below the grayscale threshold value. As noted previously, grayscale typically varies from 0 
(white) to 255 (black), however, the grayscale threshold can be expressed in a percent from 

14 
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0% (white) to 100% (black). This step effectively reduces any remaining residual 

background areas. 

A further technique to solve the problem of residual background areas, comprises 
application of a minimum pixel filter step 1 12. The minimum pixel filter step 1 12 searches 

5 for continuous black pixel regions where the number of pixels equals a number less than the 
minimum pixel filter value, and then removes these pixel regions. This allows removal of 
small, high contrast regions appearing in the background of the digitized optical section 32. 
While the default for the minimum pixel filter value comprises twenty-five, most of the 
outlined background consists of groups of pixels of between five and ten pixels. Typically, a 

10 minimum pixel filter value of between five and ten will allow for the removal of these 

unwanted background objects without interfering with the outline 38 of the digitized optical 
section 32. In a similar manner. Fig. 3 shows a maximum pixel filter step 11 4. The 
: . . maximum pixel filter step 1 1 4 allows for the elimination of large unwanted areas that appear 
within the digitized optical section 32. The maximum pixel filter step 1 14 selects those 

15 regions of the digitized optical section 32 with continuous pixel groupings above the 

maximum pixel filter size. The default maximum pixel filter value equals twenty thousand, 
but of course, will vary based on the specific application. 

Further image processing comprises the erode step 1 15 (Fig. 3). Eroding the dilated 
outline 128, 130 creates an eroded outline 132 and an eroded outline 134 respectively (Fig. 

20 6). The necessity of the erode step 1 1 5 results from the prior dilate step 106 and smoothing 
steps 102, 106 which all serve to fatten the outline 38. Therefore, to return the outline 38 to 
the proper size requires eroding the outline 38 by the number of dilation steps plus the 
number of times the outline 38 is smoothed. Referring to Fig 6, the erosion process moves 
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each pixel of the dilated outlines 128, 130 inward a distance of one pixel. In this manner, the 
eroded outlines 132, 134 now more accurately reflect the periphery of the object in the 
digitized optical sections 1 1 7, 1 18. In the preferred embodiment of the invention the dilate 
default equals three, since the erode default equals two and the smooth outline default equals 
5 one. 

Fig. 9 shows a further illustration of the result of outlining. Fig. 9 shows a plurality of 
digitized optical sections 32, each taken at a different focal depth, and the associated outline 
38 of each digitized optical section 32. In this case, not only do the outlines 3S change in size 
and shape, but some of the outlines 32 contain more than one distinct circumscribed area. 

10 Those of ordinary skill in the art will appreciate the fact that optimizing the outlining 

parameters comprises a trial and error process, that involves varying not only the outlining 
parameters but the number of times each step is performed. After selecting the optimum 
. imaging parameters, however, all of the plurality of digitized optical sections 32 can be 
processed with the optimized parameters. To illustrate the optimization process, Figs. 19-21 

15 show the effect of varying the number of times the smooth image step 102 is performed. In 
Fig. 19 the smooth image step 102 is performed once, in Fig. 20 the smooth image step 102 is 
performed twice, and in Fig. 21 the smooth image step 1 02 is performed four times. 
Increasing the smoothing of the image effectively reduces the sharpness of the image, and, 
therefore, reduces the complexity of the digitized optical section 38. This reduces the area of . 

20 the outline 38 since the smoothing reduces the contrast of the digitized optical section 38. 

Figs. 22-24 show the effect of differing combinations of the dilate step 106 and the erode step 
1 15. In Figs. 22-24 the smooth image step 102 is performed once, and the smooth outline 
step 108 is performed three times. In Fig. 22 the dilate step 106 is performed twice, and the 
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erode step 1 15 is not performed. In Fig. 23 the dilate step 106 is performed three times, and 
the erode step 1 15 is performed six times. In Fig. 24 the dilate step 106 is performed three 
times, and the erode step 1 15 is performed eight times. The overall effect shown in Figs. 22- 
24 comprises increasing the gap between the number of dilate steps 106 and the number of 
5 erode steps 1 1 5, which in general reduces the size of the outline 38. Also, increasing the 
number of dilate steps 106 and erode steps 115 between the values depicted in Fig. 22 and 
Fig. 23 helped to better fill in a particularly bright portion of outline 38. The preceding 
examples of the effect of altering the outlining parameters merely demonstrate the type of 
iterative process required for optimization, and illustrates some general trends applicable to 

1 0 changing certain parameters. The specific effect, of course, will vary depending on the exact 
circumstances of the application. 

Despite the overall effectiveness of the automatic outlining method, some instances . 
may require manual outlining. For example, Fig. 7 shows an outline 38 with a lateral 
indentation 78. The outline 38 represents the ideal, or perfect, outline 38. Applying the 

15 above outlining parameters could result in filing in the lateral indentation 78 with outline 76 
(shown in phantom). In this type of situation the 3D-DIAS system 10 provides for the 
possibility of manual outlining. After optically sectioning the object, digitizing the 
optical sections, and outlining the digitized optical sections 32, the next step comprises 
reconstructing from the plurality of digitized optical sections 32 a three dimensional graphical 

20 reconstruction of the object for computerized viewing. The 3D-DIAS System 10 

contemplates two types of reconstructions. First, a three dimensional elapsed time stacked 
image reconstruction 34 shown in Fig. 10b. Second, a three dimensional elapsed time faceted 
image reconstruction 36 shown in Fig. 10c. The stacked image reconstruction 34 essentially , 
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comprises stacking each of the digitized optical sections 32, wherein the focal depth of the 

digitized optical sections 32 translates into a height. Fig. 10a shows a plurality of twelve 

digitized optical sections 32 each at a different focal depth. The computer 22, again under the 

control of programming means, constructs a stacked image reconstruction 34 by stacking 

5 each of the digitized optical sections 32 by height. The first stacked image reconstruction 34 

of Fig. 10b shows the digitized optical sections from a 0° viewing attitude, with each 

digitized optical section labeled from one to twelve. Thus, the digitized optical section 32 

appearing in Fig. 1 0a ( I ) appears at the bottom of the stacked image reconstruction 34 shown 

in Fig. 10b at 0°, and the digitized optical section 32 appearing in Fig. 10a (12) appears at the 

10 top of the same stacked image reconstruction 34. The stacked image reconstruction 34 
viewed from the 0° viewing attitude only displays a side view of each digitized optical 
section 32, but clearly shows the height spacing between each digitized optical section. 32. 

Each stacked image reconstruction 34 displays only that portion of each of the 
plurality of digitized optical sections 32 defined by the outline 38 of the digitized optical 

15 sections 32, and visible from the particular viewing attitude. The 30° stacked image 

reconstruction 34 of Fig. 10b shows the digitized optical sections 32 of Fig. 10a viewed from 
a viewing attitude of 30° above the horizontal. In this manner, the edges of the digitized 
optical sections 32 overlap each other clearly showing the three-dimensional nature of the 
stacked image reconstruction 34. The stacked image reconstructions 34 essentially comprises 

20 overlapping a series of two dimensional digitized optical sections 32, and then displaying 
only that portion of the digitized optical sections 32 not overlapped or hidden by an 
underlying digitized optical section 32. For example, starting with the lowest level digitized 
optical section 32 shown in Fig. 10a (1), each subsequent digitized optical section 32 stacks 
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over the top of the previous digitized optical section 32. The computer assigns a grayscale 

value to each point of each of the plurality of digitized optical sections 32, with the grayscale 

of each digitized optical. section 32 decreasing by height. As each digitized optical section 32 

is laid over the lower digitized optical section 32, that portion of the proceeding digitized 

5 optical section 32 overlapped by the newly applied digitized optical section 32 is no longer 

visible from that particular viewing attitude. Fig. 10b also shows the same stacked image 

reconstruction 34 displayed from a 60° viewing attitude and a 90° viewing attitude, which 

expose for viewing different portions of the digitized optical sections 32. By creating a 

stacked image reconstruction 34 for each period of time of optical sectioning, and displaying 

10. each stacked image reconstruction 34, the 3D-DIAS System 10 creates and displays a three 
dimensional elapsed time stacked image reconstruction of the object. 

Fig. 10c shows a faceted image reconstruction 36 of the plurality of digitized optical 

. ■ . . sections 32 appearing in Fig. 10a. The facet image reconstruction method begins by. 

constructing a top wrap 84 and a bottom wrap 86 (see also Fig. 15 ). Conceptually, the top 

15 wrap 84 is essentially identical to the stacked image reconstruction 34 shown in Fig. 10b 
viewed from a 90° attitude, and the bottom wrap 86 consists of the same stacked image 
reconstruction 34 viewed from a minus 90° attitude. In other words, in abstract terms the 
faceted image reconstruction 36 consists of dividing the stacked image reconstruction 34 into 
a top wrap 84 and a bottom wrap 86. Then by casting a fishing net over the top wrap 84, the 

20 webbing of the fishing net forms facets 94 that define the outer perimeter of the top wrap 84. 
In an identical manner, the process repeats for the bottom wrap 86. The last step involves 
joining the top wrap 84 and the bottom wrap 86 at a seam. This process essentially creates a 
mathematical 3-D model of the enclosed object. 
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In more particular terms, the process of creating the top wrap 84 involves the 
following steps. First, assigning each of the plurality of digitized optical sections 32 a height 
corresponding to the focal depth. This process is identical to the process of assigning heights 
used to create the stacked image reconstructions 34. Next, the computer 22 of the 3D-DIAS 
5 System 10 identifying pixels corresponding to only that portion of the area of each of the 
plurality of digitized sections 32 defined by the outline 33, and not overlapped by another 
digitized optical section as viewed from the top of the reconstruction. Essentially, this 
involves creating the stacked image reconstruction 34 of Fig. 10b viewed from the 90° 
attitude. Thus, the digitized optical section 32 shown in Fig. 10a at (12) appears at the top of 

1 0 the stack, and the digitized optica] section 32 shown in Fig. 10a at (1 1) appears directly 
underneath. However, that portion of the digitized optical section 32 of Fig. 10a shown at 
(11), overlapped by the digitized optical section 32 shown in Fig. 10a at (12) does not appear. 
The process repeats until. the appropriate portions of each of the plurality of digitized. optical ~ 
sections 32 appears in the top wrap 84. 

15 Next, each identified pixel is assigned an X, Y and Z coordinate, where the X and Y 

coordinate correlates to the pixels row and column and the Z coordinate represents the height 
of the location of the pixel in the faceted image reconstruction 36. if the particular pixel 95 
happens to lie exactly on the outline 38 of a particular digitized optical section 32 (see Fig. 
11), then the Z coordinate equals the height of that particular digitized optical section 32. 

20 Using the analogy of a contour map, the pixels that lie directly on an outline 38 of a digitized 
optical section 32 essentially lie exactly on a contour line. This allows for quickly 
determining the Z coordinate for these particular pixels. For pixels, like pixel 96, lying 
within a particular outline 38, but not actually on the outline 38, the Z coordinate is assigned 
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a height based on a weighted average. For those pixels lying outside of the outline, for 
example background pixels, the Z coordinate of these pixels can be designated an easily 
recognizable arbitrary number like one million. 

Those skilled in the art will realize that a number of techniques can accomplish the 
5 calculation of the weighted average. One such technique, however, contemplated by the 
present invention involves defining a plurality of rays extending from each of the pixels 
within any of the outlines 38 of any of the digitized optical sections 32 to the next nearest 
outline 38. Additionally, the weighting scheme involves weighting the shortest rays more 
than the longest rays. In particular, Fig. 1 1 shows a plurality of digitized optical sections 32 

10 stacked according to height, where the height corresponds to the focal depth of the particular 
digitized optical section 32. In this manner, the plurality of digitized optical sections 32 take 
on the look of a contour map. Fig. 1 1 shows a pixel 96 located between a zero micron 
contour level 66, a plus one micron contour level 68, and a plus two micron contour level 70. 
Since the pixel 96 does not lie directly on any of the outlines 38, the Z coordinate of the pixel 

15 96 must equal a value somewhere between the heights of the surrounding outlines 38 of the 
digitized optical sections 32. One method to calculate the Z coordinate value of the pixel 96 
involves drawing a plurality of rays from the pixel 96 to the surrounding outlines 38 and 
weighting the shorter rays more than the longer rays. Fig. 1 1 shows eight rays extending at 
45° angles from the pixel 96. A first ray 50, a second ray 52, a third ray 54, a fourth ray 56, a 

20 fifth ray 58, and a sixth ray 60 all extend from the pixel 96 to the zero micron contour level 
66. A seventh ray 62 extends from the pixel 96 to the plus one micron contour level 68, and 
an eighth ray 64 extends from the pixel 96 to the plus two micron contour level 70. In this 
manner, each of the eight rays 50-64 extends a certain length of L1-L8, and contacts an 
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outline 38 of a particular height of H1-H8. Calculation of the Z coordinate of the pixel 96 
proceeds by using an equation that weights the heights H1-H8 in an inverse proportion to 
their lengths L1-L8, in the following manner: 

5 Z = HiO/L ^nO/^ 

" (1/L~) + (\/L{) + (1/L 3 ) + (1/L 4 ) + (I/L5) + ("1/Lftf + (1/L 7 ) + (1/Lg) 

This process repeats until each pixel of the top wrap 84 is assigned a X, Y and a Z 
coordinate. In a converse fashion, the computer 22 of the 3D-DIAS System 10 constructs a 

10 bottom wrap 86. The only difference between the bottom wrap 86 and the top wrap 84 

involves the viewing attitude of the stacked image reconstruction 34. The top wrap 84 uses a 
viewing attitude of 90°, while the bottom wrap 86 repeats the process with a viewing attitude 
of minus 90°. In other words, the top wrap 84 views the stacked image reconstruction 34 
from the top, and the bottom wrap 86 views the stacked image reconstruction 34 from the 

15 bottom. 

After creating the bottom wrap 86, and the top wrap 84 the next step in the method 
comprises dividing the bottom wrap 86 and the top wrap 84 into facets, where the facets 94 
(see Fig. 15) encompass the surface area of the top and bottom wraps 84, 86. The facets 94 
comprise the area between the vertical and horizontal contour lines that surround the top and 
20 bottom wraps 84, 86. The vertices of the facets 94 have X, Y and Z coordinates defined by 
the above process, and are, therefore, easily quantified. While Fig. 1 5 shows the facets 94, 
Fig. 10c provides a better illustration but the facets are too numerous to provide for 
numbering. Fig. 10c shows the faceted image reconstruction 36 of the digitized optical 
sections 32 shown in Fig. 10a, viewed from a plurality of viewing attitudes. The faceted 

22 

SUBSTITUTE SHEET (RULE 26) 



WO 99/67739 PCT/US99/I3193 
image reconstruction 36 of Fig. 10c viewed at an attitude of 90° shows the facets 94 of the 
top wrap 84. Dividing the top wrap 84 and the bottom wrap 86 according to vertical and 
horizontal contour lines creates the facets 94 at the intersections of the contour lines. 
Therefore, the perimeter of each facet 94 is defined by a pixel with a X, Y and Z coordinate. 
5 By evaluating the X, Y and Z coordinates of the pixels of the top wrap S4 and the bottom 
wrap 86, allows identification of a seam which defines the intersection of the facets 94 of the 
top wrap 84 and the facets 94 of the bottom wrap 86. Joining the top wrap 84 and the bottom 
wrap 86 at the seam allows creation of the faceted image reconstruction 36, and repeating this 
process over several periods of time allows creation of a three dimensional elapsed time 

10 faceted image reconstruction. 

One difficulty encountered with the faceted image reconstruction process involves the 
inability to accurately depict advanced and complicated contours. Fig. 1 5 shows a stacked 
image reconstruction 34 of a plurality of a digitized optical sections 32. which contains a - 
pronounced lateral indentation 78. For pronounced lateral indentations, like the lateral 

15 indentation 78 of Fig. 1 5, the faceted image reconstruction method will not accurately 

describe the area of indentation. Digitized optical sections 32 above-and below the lateral 
indentation 78 overhang the lateral indentation 78. Again, referring to the fishing net 
analogy, casting a net over the stacked image reconstruction 34 will not completely define the 
surface area defined by the digitized optical sections 32. The solution to this problem 

20 involves identifying the lateral indentation 78 at the maximum point of advance, and then 
subdividing the object at the maximum point of advance creating a top partial wrap 80 and a 
bottom partial wrap 82. Next, the method proceeds by performing the aforementioned steps 
of creating the faceted image reconstruction 36 on the top partial wrap 80, and the bottom 

23 



SUBSTITUTE SHEET (RULE 26) 



WO 99/67739 PCTAJS99/1 3 1 93 

partial wrap 82. Identification of the lateral indentation 78 generally requires manual 
intervention, wherein the necessity of identifying the lateral indentation 78 will depend on the 
. particular circumstances and the contour of the particular object involved. Joining the top 
partial wrap 80 and the bottom partial wrap 82 at their seam results in creation of a partial 
5 faceted image reconstruction 98. The partial faceted image reconstruction 98 clearly shows 
the lateral indentation 78. Since the lateral indentation 78 could appear in either the top wrap 
84 or the bottom wrap 86, the process of creating the partial faceted image reconstruction 98 
merely involves dividing either the top wrap 84 or the bottom wrap 86 at the lateral 
indentation 78, and then separately processing the top partial wrap 80, and the bottom partial 

10 wrap 82. This process, of course, can repeat in order to define successive lateral indentations. 
Some situations may require tracking the motility and morphology of an interior 
portion of the moving object. Fig. 12 shows an example of such a situation. Fig. 12 shqws a 
stacked image reconstruction 34. of a cell over a plurality of time periods. Each of the stacked 
image reconstructions 34 of the cell contain a slot 40, representing the location of the nucleus 

15 of the cell. Below each of the stacked image reconstructions 34 appears a faceted slot 42 

representing the nucleus of the cell. Fig. 13 shows a single digitized optical section 32 with a 
slot 40, and a slice 74 which divides the digitized optical section 32 and the slot 40 into two 
portions. Creating the stacked image reconstruction 34 of Fig. 12 involves outlining each 
digitized optical section 32, identifying a slot 40 in each of the digitized optical sections 32, 

20 and dividing each slot 40 of each digitized optical section 32 at a slice 74. The stacked image 
reconstruction 34 involves stacking one of the portions of the digitized optical sections 32 
defined by the slice 74. This allows viewing both the stacked image reconstruction 34 and 
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the slot 40 in the same image. Outlining the slot 40 can involve the aforementioned 

automatic outlining process; or can proceed manually. 

Fig. 14 shows an example of a plurality of faceted image reconstructions 36 over a 

period of time including a first faceted slot 44, a second faceted slot 46, and a third faceted 

5 slot 48. Fig. 14 shows the faceted image reconstruction 36 at seven different time periods, 

and from two different viewing attitudes. The top group of faceted image reconstructions 36 

appears at a 0° viewing attitude, while the bottom group of faceted image reconstructions 36 

appears at a 90° viewing attitude. In this manner, Fig. 14 shows that the method of the 

present invention can depict the motility and morphology of not only a moving object, but of 

1 0 selected portions of the moving object. 

The reconstruction methods of the present invention provides a three dimensional 
mathematical model for computing motility and dynamic morphology parameters. Fig. 8 
shows an example of a graphical user interface screen to allow a user to select from a — 
plurality of parameters representing the motility and morphology of an object. Calculation of 

1 5 parameters representing the motility and dynamic morphology of an object requires defining 
the following notation: 
Notation: 

U F" equals the total number of digitized optical sections involved in the calculation, 
while "f "equals the digitized optical section subject to the current calculation; 
20 "X[f],Y[fj" equals the coordinates of the centroid of digitized optical section f, 

where 1< f < F; 

"I" equals the centroid increment and defines what previous and subsequent mean (for 
example a centroid increment of I means the centroid based calculations of the 
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N'th digitized optical section use the N-I previous digitized optical section and 
the N+I subsequent digitized optical section), increasing the centroid 
increment tends to smooth the particular value, and reduces sudden uneven * 
jumps; 

"iT equals the number of pixels in a digitized optical section's outline, where P, . . . 

P N represents the n individual pixels, and where P xn and P yn comprises the X 

and Y coordinates of the n'th pixel; 
"frate" equals the number of digitized optical sections per unit of time; 
"scale" equals the scale factor in distance units per pixel; 
"sqrt[number]" returns the square root of the number; 
"abs[number]" returns the absolute value of the number; 

"angle[X, Y] M returns the angle in degrees between a vector with origin (X v Y) and the 

X axis, with positive angles measured counter-clockwise; - .- 
"NAN" equals NOT A NUMBER, an arbitrary large designation (1,000,000 for 

example) generally used to indicate a non-processed value; and 
"Central Difference Method" (CDM), CDM calculations use the previous and 

subsequent centroids in the calculation, while non CDM calculations use only 

the pervious centroid. 

Parameters: 

Speed: 

For(f-Kl), 

Speed[f] = 0; 
For(f-I> 1), 
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Speed[f] = (scale)(frate)(sqrt[ ((X[f] - X[f-I])/I) 2 + ((Y[f] - Y[f-I])/I) 2 

Speed (CDM): 

For(f-I> l)and(f+I<F), 



Speed[f] = (scale)(frate)(sqrt[ ((X[f+I] - X[f-I])/I) 2 + ((Y[f+I] - Y[f- 



5 I])/I) 2 



For(f-I< l)and (f+I<F), 

Speedff] = (scale)(frate)(sqrt[ ((X[f+I] - X[f])/I) 2 + «Y[f+I] - Y[f])/I) 2 
For(f-I> l)and(f+I>F), 

Speed[f| = (scale)(frate)(sqrt[ ((X[f) - X[f-I])/I) 2 + ((Y[f] - Y[f-I])/I) 2 
10 For all other f, 

Speed[f] = 0 

Direction: 

For(f-T> 1 ), — ' •• - • - 

Dir[f] = angle[ (X[f] - X[f-I]) , (Y[f] -Y[f-I) ] 
15 For(f- 1< l)and(f+I<F), 

Dir[f] = anglef (X[f+I] - X[f]) , (Y[f+1] -Y[f]) ] 
For all other f, 

Dir[f] = 0 
Direction (CDM): 
20 Fdr(f-I>l)and(f+I^F), 

Dir[f] = angle[ (X[f+I] - X[f-I]) , (Y[f+I] -Y[f-I) ] 
For(f- 1 < l)and(f+I<F), 

Dirff] = angle[ (X[f+I] - X[f]) , (Y[f+I] -Y[f]) ] 
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For (f - 1 > 1) and (f + I > F), 

Dirffl = angle[ (X[f] - X[f-I]) , (Y[f] -Y[f-I]) ] 

For all other f, 

Dir[f] = 0 

Note - Multiples of ± 360° add to the direction to make the graph continuous. 
For example, on object moving in a spiral would have directions: 0°, 45°, 90°, 
135°, 180°, 225°, 270°, 315°, 360°, 405°, ect. 
Direction Change: 

For (f - 1 < I), 

DirCh[fj = 0 
For all other f, 

. DirCh[fJ = Abs[ Dir[f] - Dir[f-I] ] 
Note - if the direction changes is greater than 180° it is subtracted from 360°. 
This always gives values between 0° and 180°. 
Acceleration: 

For(f-I>l), 

Acc[f] = Speed[fJ - Speed[f-I] 
For all other f, 

Acc[fj = 0 
Acceleration (CDM): 

For (f-I> l)and(f+I<f), 

Acc[fj = (Speed[f+I] + Speed[f-I])/2 
For(f-Kl)and(f+I< F), 
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Acc[f] = (Speed[f+I] + Speed[f])/2 
For(f-I> l)and (f+I>F), 

Acc[f] = (Speedff] + Speed[f-I])/2 
For all other f, 

Acc[f] =0 - 

Persistence: 

Persis[f] = Spced[f]/(( I + 1 00/360)(DirChg[f])) 
Note - Persistence is essentially speed divided by the direction change 
(converted from degrees to grads). One is added to the denominator to prevent 
division by 0. If an object is not turning the persistence equals the speed. 
Centroid: 

CenX[f] = £/V/n : : * 



Note - To convert the centroid to a meaningful number requires multiplication 
by the scale factor. 



This first requires defining the Major Axis of the digitized optical 
section. This involves finding the pixel furthest from the centroid, this pixel 
becomes the first end point of the major axis. The second end point of the 
major axis comprises the pixel furthest from the first end point of the major 
axis. Thus, the major axis equals the chord connecting the first end point to 
the second end point. 




Axis Tilt: 
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Tilt[f] = angle in degrees between the major axis and the horizontal axis 

Note - Multiples of ± 180° are added to the axis tilt continuity. In this case 

divide the axis tilt by 1 80 and take the remainder. Thus, the graph of axis tilt 

versus time for an oblong object spinning at a constant rate will have a 

constant positive slope for a counter-clockwise spin. 

Mean Width: 

Mean Width[f) = Area[f]/MaxLen[f] 

Maximum Width: 

MaxWid[f] = length of the longest chord perpendicular to the major axis 
Central Width: 

CenWid[f] = length of the chord perpendicular to the major axis and passing 
though the centroid 

X Bounded Width: ... . 

XWid[f] = width of the smallest rectangle enclosing the digitized optical 
section's outline 

Maximum Length: 

MaxLen[f] = length of the major axis 

Y Bounded Width: 

YWid[f] = height of the smallest rectangle enclosing the digitized optical 
section's outline 

X Slice Width: 

XSWid[f] = the length of the longest chord parallel to the XWid[f] 

Y Slice Width: 

30 



SUBSTITUTE SHEET (RULE 26) 



WO 99/67739 PCT/US99/13 1 93 

YSWidff] = the length of the longest chord parallel to the YWidff] 

Area: 

Where Area equals the area of the outline of the digitized optical 
section shape minus any holes. Let X[i],Y[i] for i=0 ... n be the vertices of 
the outline such that X[0] = X[n] and Y[0] = Y[n] (the first vertex is the last 
vertex). Further, let dx[i] = X[i+1] - X(i] and dy[i] = Y[i+1] - Y[i]. Then by 
Green's Theorem the area is the following. 

Areaff] = 0.5(abs[ Y(x[i))(ciy[i]) - (y[i)dx[i)) )) 
Perimeter: 

The perimeter equals the perimeter of the outline of the digitized 
optical section plus the perimeter of any holes. Let X[i],Y[i] for i=0 ... n be 
the vertices of the outline such . that X[0] = X[n] and Y[0] = Y[n] (the first 
vertex is the last vertex). Further, let dx[i] = X[i+1] - X[i] and dy[i] = Y[i+1] - 
Y[i]. Then the perimeter if the following. 

Perimeterff] = ^sqr%dx[i] 2 + </v[/] 2 ] 

1=0 

Roundness: 

Round[f] = (100)(47r)(Area[f]/Perim[f] 2 ) 

Roundness is a measure (in percent) of how efficiently a given amount 
of perimeter encloses an area. A circle has roundness of 1 00%, while a 
straight line has a roundness of 0%. The factor of 4ti in the formula ensures a 
roundness values of 100% for a circle. The perimeter is squared to make the 
roundness invariant (i.e. dimensionless). 
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Predicted Volume: 

Vol[f] « (47t/3)(MaxLen[f]/2)(MeanWid[f]/2r 

The predicted volume Vol[f] is the volume of the ellipsoid, with 

circular cross-section, having length MaxLen[f] and width Mean Wid[f], 
Predicted Surface: 

Surff] = (CF)(7t)(MaxLen[f])(MeanWid[f|) 

The predicted surface area Sur[f] equals the surface area of the 

ellipsoid, with circular cross-section, having length MaxLenff] and width 

MeanWid[f]. Where CF is the ellipsoidal surface correction factor defined by 

CF = U (sin[X])(sqrt[ sin 2 [X] + (r)(cos 2 [X])J)dx 

Where r = MeanWid[f]/MaxLen[f). Using Simpson's Rule with N=10000, the 
computer approximates the solution of CF with the following polynomial. 
CF= 0.15r + 0.065r + 0.785 
Mean Radial Length: 

The mean radial length RadLen[f] is the average distance from the 
centroid to the boundary pixels. Let-n be the number of vertices (equal to the 
number of boundary pixels) of the digitized optical section's outline, indexed 
from 0 to n-1. Let L[i] equal the distance from the i'th vertex to the centroid. 
Then 



RadLen[f] = £z[i]/n 
Radial Deviation: 



r=0 
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The radial deviation RadDev[f] equals the ratio of the standard 

deviation of the above average to that average itself in percent. Let SD equal 

the standard deviation of L[0] L[n-1], 

RadDev[f] = (100)SD/RadLen[f] 

Convexity and Concavity: 

To compute Convexff] and Concavff] requires drawing line segments 

connecting each vertices of the outline. The angles of turning 116 from on 

segment to the next are measured (Fig. 17). Counter-clockwise turning 

represents a positive angle, while clockwise turning a negative angle. For a 

closed outline, these angles always add up to 360°. The procedure repeats for 

holes in the outline. 

Convexff] = sum , of the positive turning angles . . 
Concav[f] = abs[ sum of the negative turning angles] 
Also, Convex[f] -Concavff] = (360)(1 + Number of Holes). 

Convexity and concavity measure the relative complexity of the shape 
of the outline. For example, the convexity of a circle equals 360 and the 
concavity equals 0. 
Positive and Negative Flow: 

Positive flow essentially measures the amount of new area formed in a 
certain amount of time (or in the flow increment), expressed in percent. 
Conversely, negative flow measures the amount of area lost over the period of 
time designated by the flow increment in percent. In other words, positive and 
negative flow measure the percent of area expansion and contraction of an 
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object over a period of time. In particular, let f equal the current frame and FI 

equal the flow increment. Let A equal the interior of the f-FI outline, minus 

any holes, and B equal the interior of the fth outline, minus any holes (with 

positive and negative flow undefined for f-FI < 1 ). Furthermore, let P equal 

the area in B not present in A, or P = B - A. Let N equal the area in A not 

present in B, or N = A - B. Then 

PosFlow[f]= (100)Area(P)/Area(A) 

NegFlow[f]= (100)(Area(N)/Area(A) 

An additional option for calculation of flow involves fixing the 

centroids over the flow increment. This aligns the B and A area so that the 

centroids overlap prior to computing flow, and subtracts out centroid 

movement from the shape change. 



Sectors: 

Sector Area, Sector Perimeter, Sector Positive Flow, and Sector 
. Negative Flow comprise derivative measurements of the respective standard 
parameters. The sector measurements allow parametization of a subset, or 
sector, of a particular outline. The user inputs the beginning and ending flow 
in degrees, and the flow range is divided into four sectors. For example, 
entering 0 and 360 will produce four sectors with sector 1 consisting of 0° to 
90°, sector 2 consisting of 91° to 180°, sector 3 consisting of 181° to 270°, and 
sector 4 consisting of 271° to 360°. 
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The following summarizes a number of three dimensional parameters representing the 

motility and dynamic morphology of an object: 

3D Centroid: 

An average of all the X coordinates of each facet vertex, average of all 
of the Y coordinates of the each facet vertex, and the average of all of the Z 
coordinates of each facet vertex. 

3D Surface Area: 

The sum of all the surface areas of each facet. 

3D Volume: 

This involves first converting each facet into a prism by extending the 
facet inward to the centroid. The volume then equals the sum of the volumes 
of each prism. 

3D Height: : • .7/ 

The difference between the highest and lowest Z coordinate. 

3D Bulk Height: 

In the case where the highest or lowest Z coordinate comes from an 
extruding thin tendril of the object, the bulk height might yield more 
meaningful information. The first step comprises setting a 3D volume 
threshold in percent. The 3D Bulk Height equals the 3D Height after 
eliminating a portion of the 3D Volume equal to the threshold percentage. 
3D Length: 

The longest chord extending through the centroid from one facet to another. 
3D Width: 
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The longest chord coming from the disc defined by all chords passing 

through the centroid and perpendicular to the 3D Length. 

Sphericity: 

The 3D analog of roundness, essentially a measurement of the 
efficiency of enclosing the 3D Volume with the 3D Area in percent. The 
Sphericity essential comprises an invariant ratio of the area to the volume. 
The sphericity of a perfect sphere would equal 100%. 
Overhang: 

This would measure the amount any portion of the object overhangs a 
base of the object, in terms of a ratio scaled to from 0 (no overhang) to 100 
(maximum overhang). For a given stacked image reconstruction, let A equal 
the width of the base digitized optical section's outline as viewed from a given 
attitude. Let B equal the width of the widest digitized optical section's outline. 
Then the Overhang equals the ratio of B:A. 
Area of Projection: 

The Area of Projection equals the ratio to the length of the digitized 
optical section's outline with the greatest area to the length of the base 
digitized optical section's outline. 
Those of ordinary skill in the art will realize the possibility of converting any number 
of the two dimensional parameters into three dimensional parameters. For example, all of the 
centroid based parameters easily convert to three dimensional parameters by substituting the 
3D Centroid. Additionally, the Graphical User Interface allows for the plotting of graphs of 
each of the parameters. Fig. 18 shows an example of a graph of Speed versus time. 
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Fig. 16 shows an alternative embodiment of the 3D-DIAS Computer System 10 

configured for parallel processing. The configuration involves connecting ten state of the art 

parallel processors 90 through a network connection. The parallel processors 90 can 

comprise power PC based Macintosh clones communicating over a vast ethemet network 

5 (two megabytes per second transfer rate) or the accelerated SCSI ports (ten megabytes per 

second). It is anticipated that by utilizing 225 megahertz power PC based computers 

connected by ethernet the 3D-DIAS System 10 can accomplish near real time reconstruction 

and analysis. In the configuration shown in Fig. 16. a distribution work station 88 controls 

the frame grabber 18. The ten parallel processors 90 perform the steps of outlining, and an 

10 integration work station 92 integrates the information from the parallel processors 90 and 

generates the reconstructed images. The reconstructed image is then played as a dynamic 3D 
image with four superimposed mini screens which display selected parameters on the 
. .computer display terminal 26. The parallel processing system utilizes software which 
includes a program for separating the digitized optical sections 32 between the parallel 

1 5 processors 90, and software for reintegrating the information from the parallel processors 90 
in the integration work station 92. 

The foregoing detailed description of the present invention comprises an exemplary 
embodiment. An additional embodiment, included herewith in the form a micro-fiche deposit 
of the 3D-DIAS software source code, will vary in some specific aspects of implementation 

20 with out departing from the scope of the present invention. The foregoing description and 
drawings comprise illustrative embodiments of the present inventions. The foregoing 
embodiments and the methods described herein may vary based on the ability, experience, 
and preference of those skilled in the art. Merely listing the steps of the method in a certain 
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order does not constitute any limitation on the order of the steps of the method. The 

foregoing description and drawings merely explain and illustrate the invention, and the 

invention is not limited thereto, except insofar as the claims are so limited. Those skilled in 

the art who have the disclosure before them will be able to make modifications and variations 

therein without departing form the scope of the invention. 
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A method for three dimensional elapsed time analysis of the motility and morphology 

of a moving object, said method comprising: 

a) optically sectioning an object at a plurality of focal depths over a first period 
of time; 

b) digitizing each of said plurality of optical sections; 

c) identifying with each of said plurality of digitized optical sections a tag that 
allows identification of at least said time and said focal depth of each of said 
plurality of digitized optical sections; 

d) outlining the periphery of said object for each of said plurality of digitized 
optical sections; 

e) calculating from said plurality of digitized optical sections a plurality of 
parameters representing the motility and morphology of said object; 

0 - - reconstructing from said plurality of digitized optical sections a three. 

dimensional graphical reconstruction of said object for computerized viewing; 
and 

g) repeating said steps of said method for at least one other period of time. 

The invention in accordance with claim 1 further comprising displaying said graphical 
representation with a pseudo-three-dimensional viewing means. 

The invention in accordance with claim 1 further comprising the step of windowing 
said plurality of optical sections whereby only a portion of interest of said plurality of 
optical sections is digitized. 
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The invention in accordance with claim 1 wherein said object is optically sectioned by 

differential interference contrast microscopy. 



The invention in accordance with claim 1 wherein said optical sections are digitized 
by a frame grabber. 

The invention in accordance with-claim 1 wherein said optical sections are digitized 
by a digital camera. 

The invention in accordance with claim 1 wherein said outlining utilizes a complexity 
algorithm. 

The invention in accordance with claim 1 wherein said outlining, further comprises: 

a) applying a complexity algorithm to each pixel of said digitized optical section 
thereby removing said pixels below said complexity threshold; 

b) dilating each of said remaining pixels of said digitized optical section; 

c) smoothing each of said remaining pixels of said digitized optical section; 

d) applying a grayscale threshold to each of said remaining pixels thereby 
removing said pixels below said grayscale threshold; 

e) applying a minimum pixel group filter thereby removing any pixel groups 
below said minimum pixel group size; 

f) applying a maximum pixel filter thereby removing any pixel groups above 
said maximum pixel filter; and 

g) eroding said remaining pixels of said digitized optical section. 
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The invention in accordance with claim 8 further comprising eroding each of said 
remaining pixels of said digitized optical section a number of times equal to the 
number of said dilating steps plus the number of said smoothing steps. 

The invention in accordance with claim 1 wherein said graphical representation of 
said object comprises creating a three dimensional elapsed time stacked image 
reconstruction. 

The invention in accordance with claim 10 wherein the steps of creating said three 
dimensional elapsed time stacked image reconstruction comprise: 

a) selecting an attitude for viewing said reconstruction; 

b) assigning said plurality of digitized optical sections a height corresponding to. 
said focal depth as viewed from said attitude; and 

c) identifying pixels corresponding to only that portion of an area of each of said 
plurality of digitized optical sections defined by said outline and not 
overlapped by another of said plurality of digitized optical sections, as viewed 
from said viewing attitude. 

The invention in accordance with claim 1 wherein said graphical representation of 
said object comprises creating a three dimensional elapsed time faceted image 
reconstruction. 
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The invention in accordance with claim 12 wherein said steps of creating said three 

dimensional elapsed time faceted image reconstruction comprises: 

a) creating a top wrap, said steps of creating said top wrap comprising: . 

i) assigning said plurality of digitized optical sections a height 
corresponding to said focal depth; 

ii) identifying pixels corresponding to only that portion of an area of each 
of said plurality of digitized optical sections defined by said outline 
and not overlapped by another digitized optical section, as viewed from 
a top of said reconstruction; and 

iii) assigning each of said pixels of said digitized optical sections an X, Y, 
and Z coordinate, wherein said X and Y coordinates correspond to said 

. pixel location, and for each of said pixels lying on an outline of a 
digitized optical sections said Z coordinate is assigned said height of 
said digitized optical section, and for each of said pixels within any of 
said outlines of said plurality of digitized optical sections said Z 
coordinate is assigned a height based on a weighted average; 

b) creating a bottom wrap, said steps of creating said bottom wrap comprising: 

i) assigning said plurality of digitized optical sections a height 
corresponding to said focal depth; 

ii) identifying pixels corresponding only that portion of an area of each of 
said plurality of digitized optical sections defined by said outline and 
not overlapped by another digitized optical section, as viewed from a 
bottom of said reconstruction; and 
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iii) assigning each of said pixels of said digitized optical sections an X, Y, 
and Z coordinate, wherein said X and Y coordinates correspond to said 
pixel location, and for each of said pixels lying on an outline of a 
digitized optical sections said Z coordinate is assigned said height of 
said digitized optical section, and for each of said pixels within any of 
said outlines of said plurality of digitized optical sections said Z 
coordinate is assigned a height based on a weighted average; 

c) dividing said top wrap and said bottom wrap into facets, wherein said facets 
encompass the surface area of said top and said bottom wrap; and 

d) joining said top wrap and said bottom wrap at a seam defined by the 
intersection of said facets of said top wrap with said facets of said bottom 
wrap. 

The invention in accordance with claim 13 wherein said weighted average is 
calculated by defining a plurality of rays extending from each of said pixels within 
any of said outlines of said plurality of digitized optical sections to the nearest of said 
outlines, and wherein the shortest of said rays are weighted more than the longest of 
said rays. 

The invention in accordance with claim 14 further comprising eight rays of lengths L,, 

L 2 , L 3 , L 4 , L 5 , L 6 , L 7 , and L 8 extending at 45° angles to the nearest eight outlines of 

heights H„ H 2 , H 3 , H 4 , H 5 , H 6 , H 7 , and H 8 , and said Z coordinate is defined by 

Z = H^/^ HH^/L^+H .n/M^^ 

" (1/L,) + (1/L 2 ) + (f/Lj) + (1/L 4 ) +~ (I/L5) + (1/L 6 ) + (l/L 7 ) + (l/Lg) 
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The invention in accordance with claim 13 further comprising: 

a) identifying a lateral indentation in said plurality of digitized optical sections at 
a maximum point of advance; 

b) subdividing said plurality of digitized optical sections at said maximum point 
of advance thereby creating a top partial wrap and a bottom partial wrap; and 

c) performing the steps of claim 13 on said top partial wrap and said bottom 
partial wrap. 

The invention in accordance with claim 1 further comprising creating a slot by 
outlining the periphery of a section of said object, for each of said plurality of 
digitized optical sections. 

A method for three dimensional elapsed time analysis of the motility and morphology 
of a moving object, said method comprising: 

a) optically sectioning by differential interference contrast microscopy an object 
at a plurality of focal depths over a first period of time; 

b) windowing said plurality of optical sections whereby only a portion of interest 
of said plurality of optical sections is digitized; 

c) digitizing each of said plurality of optical sections with a frame grabber; 

d) identifying with each of said plurality of digitized optical sections a tag that 
allows identification of at least said time and said focal depth of each of said 
plurality of digitized optical sections; 
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e) outlining the periphery of said object for each of said plurality of digitized 

optical sections, said outlining comprising: 

i) applying a complexity algorithm to each pixel of said digitized optical 
section thereby removing said pixels below said complexity threshold; 

ii) dilating each of said remaining pixels of said digitized optical section; 

iii) smoothing each of said remaining pixels of said digitized optical 
section; 

iv) applying a grayscale threshold to each of said remaining pixels thereby 
removing said pixels below said grayscale threshold; 

v) applying a minimum pixel group filter thereby removing any pixel 
groups below said minimum pixel group size; 

vi) applying a maximum pixel group filter thereby removing any pixel 
. .. .groups above said maximum. pixel group size; and . 

vii) eroding said remaining pixels of said digitized optical section a number 
of times equal to the number of said dilating steps plus the number of 
said smoothing steps; 

f) calculating from said plurality of digitized optical sections a plurality of 
parameters representing the motility and morphology of said object; 

g) reconstructing from said plurality of digitized optical sections a three 
dimensional graphical elapsed time faceted image reconstruction of said object 
for computerized viewing, said reconstructing comprising: 

i) creating a top wrap, said steps of creating said top wrap comprising: 
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a) assigning said plurality of digitized optical sections a height 

corresponding to said focal depth; 

b) identifying pixels corresponding only that portion of an area of 
each of said plurality of digitized optical sections defined by 
said outline and not overlapped by another digitized optical 
section, as viewed from a top of said reconstruction; and 

c) assigning each of said pixels of said digitized optical sections 

an X, Y, and Z coordinate, wherein said X and Y coordinates 

correspond to said pixel location, and for each of said pixels 

lying on an outline of a digitized optical sections said Z 

coordinate is assigned said height of said digitized optical 

section, and for each of said pixels within any of said outlines 

of said plurality of digitized optical sections said Z coordinate 

is assigned a height based on a weighted average wherein said 

weighted average is calculated by defining eight rays of lengths 

L h L 2 , L 3 , L 4 , L 5 , L 6 , L 7 , and L 8 extending from each of said 

pixels within any of said outlines at 45° angles to the nearest 

eight outlines of heights H,, H 2 , H 3 , H 4 , H 5 , H 6 , H 7 , and H 8 , and 

said Z coordinate is defined by 

Z = H^/L. HH.a/LP+H i n^^ 

" (1/L,) + (l/L 2 ) + (1/L 3 ) + (I/L4) + 0/L 5 ) + (1/LJ+ (l7L 7 ) + (1/Lg) ; 

ii) creating a bottom wrap, said steps of creating said bottom wrap 
comprising: 



47 



SUBSTITUTE SHEET (RULE 26) 



WO 99/67739 PCT/US99/ 13193 

a) assigning said plurality of digitized optical sections a height 

corresponding to said focal depth; 

b) identifying pixels corresponding only that portion of an area of 
each of said plurality of digitized optical sections defined by 
said outline and not overlapped by another digitized optical 
section, as viewed from a bottom of said reconstruction; and 

c) assigning each of said pixels of said digitized optical sections 
an X, Y, and Z coordinate, wherein said X and Y coordinates 
correspond to said pixel location, and for each of said pixels 
lying on an outline of a digitized optical sections said Z 
coordinate is assigned said height of said digitized optical 

. . section, and for each of said pixels within any of said outlines 
of said plurality of digitized optical sections said Z coordinate 
is assigned a height based on a weighted average wherein said 
weighted average is calculated by defining eight rays of lengths 
L,, L 2 , L 3 , L 4 , U, L 0 , L 7 , and L g extending from each of said 
pixels within any of said outlines at 45° angles to the nearest 
eight outlines of heights H,, H 2 , H 3 , H 4 , H 5 , H 6 , H 7 , and H 8 , and 
said Z coordinate is defined by 

Z « H, azL 1 j±H 2 q^ 

" (1/L,) + (1/L 2 ) + (f/L 3 ) + (1/L 4 ) +"(1/L 5 ) + (1/L 6 ) + (l7L 7 ) + (1/L„) ; 
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iii) dividing said top wrap and said bottom wrap into facets, wherein said 

facets encompass the surface area of said top and said bottom wrap; 
and 

iv) joining said top wrap and said bottom wrap at a seam defined by the 
intersection of said facets of said top wrap with said facets of said 
bottom wrap; 

h) displaying said graphical representation with a pseudo-three-dimensional 
viewing means; 

i) creating a slot by outlining the periphery of a section of said object, for each of 
said plurality of digitized optical sections; and 

j) repeating said steps of said method for at least one other period of time. 

An apparatus for three dimensional elapsed time analysis of the. motility and . ... 
morphology of a moving object, said apparatus comprising: 

a) a microscope for optically sectioning a motile object at a plurality of focal 
depths over a first period of time; 

b) a means for digitizing each of said plurality of digitized optical sections; 

c) a means for identifying each of said plurality of digitized optical sections with 
a tag that allows identification of said time and said focal depth of each of said 
plurality of digitized optical sections; 

d) a computer means for outlining the periphery of said object for each of said 
plurality of digitized optical sections; 

49 



SUBSTITUTE SHEET (RULE 25) 



WO 99/67739 



PCT/US99/I3193 



e) 



a computer means for calculating from said plurality of digitized optical 



sections a plurality of parameters representing the motility and morphology of 



said object; 



f) 



a computer means for reconstructing from said plurality of digitized optical 



15 



sections a three dimensional graphical reconstruction of said object for 



computerized viewing; and 



g) 



a computer means for repeating said steps of said method for at least one other 



period of time. 



20. The invention in accordance with claim 19 further comprising a stepper motor 
attached to said microscope. 

21. The invention in accordance with claim 19 wherein said microscope utilizes, digital .... 
interference contrast microscopy. 

22. The invention in accordance with claim 19 wherein said means for digitizing each of 
said plurality of optical sections comprises a frame grabber. 

23. The invention in accordance with claim 19 wherein said means for digitizing each of 
said plurality of optical sections comprises a digital camera. 

24. The invention in accordance with claim 19 wherein said means for identifying each of 
said plurality of digitized optical sections with a tag that allows identification of said 
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time and said focal depth of each of said plurality of digitized optical sections 
comprises a character generator. 

25. The invention in accordance with claim 19 wherein said computer means comprises a 
single general purpose computer. 

26. The invention in accordance with claim 19 further comprising: 

a) a first distribution workstation computer for receiving said digitized optical 
sections, said time, and said focal depth; 

b) a plurality of parallel processor computers for outlining the periphery of said 
object for each of said plurality of digitized optical sections; and 

c) an integration workstation computer for integrating said outlines, calculating 
said parameters, and Tor reconstructing said threadimensionaLgraphicai. 
representation of said object. 
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